Skip to content

Introduce a structured YAML partition source and migrate Glymur-CRD#134

Open
Igor Opaniuk (igoropaniuk) wants to merge 17 commits into
qualcomm-linux:mainfrom
igoropaniuk:feat/add-support-for-yaml-format
Open

Introduce a structured YAML partition source and migrate Glymur-CRD#134
Igor Opaniuk (igoropaniuk) wants to merge 17 commits into
qualcomm-linux:mainfrom
igoropaniuk:feat/add-support-for-yaml-format

Conversation

@igoropaniuk

@igoropaniuk Igor Opaniuk (igoropaniuk) commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

This series replaces the legacy line-format partitions.conf with a structured, schema-validated YAML source and migrates the first board, Glymur-CRD, onto it without changing a single generated artifact. contents.xml.in for glymur-crd stays intact, as it feeds a separate emitter (gen_contents), not the partition source. This work only replaced partitions.conf. Generating contents.xml from the board YAML (and thus removing the template) will be added in a next PR.

It first decouples the parser behind a LoadedSpec intermediate representation and a loaders package, then adds a YAML loader validated by a packaged JSON Schema that normalises field-for-field to the same LoadedSpec the .conf loader produces, so both formats emit byte-identical XML.

On top of that it adds a multi-storage board model with a composition resolver - board-level extends for single-base inheritance, storage-level includes for shared partition fragments, and hlos/boot-fw variant overlays, all deep-merged by stable identifier under a deterministic ordering rule - together with a show subcommand and a gen_partition --board mode that emits one partitions.xml per storage.

Finally it authors platforms/boards/glymur-crd.yaml with a Debian HLOS variant, switches the Makefile to build Glymur-CRD from that board, and drops the two now-redundant partitions.conf files; every partitions.xml, GPT binary and contents.xml is verified byte-identical to the previous output through the pinned checksum manifest.

PyYAML and jsonschema are the only new runtime dependencies. Partition order is preserved verbatim because it determines GPT offsets, which is why the shared SPI-NOR blocks stay inline rather than being factored through includes for this already-laid-out board. See [1] for the design discussion and rationale.

[1] #124

The module ran argv parsing and XML emission as top-level statements at
import time, so it could not be imported by pytest or any library
caller. This blocks the follow-up work: unit-testing the parser and
splitting it into a loaders package.

Wrap the flow in main(argv=None) -> int gated by __name__ ==
"__main__", move the module-level accumulators into locals, and return
status codes instead of sys.exit(). The runpy dispatcher in cli.py is
unaffected because it sets __name__ to "__main__".

Behaviour-preserving: every platforms/*/*/partitions.xml is unchanged.

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The .conf format was hard-coded into gen_partition.py, with no boundary
between where data comes from and how partition XML is emitted. Adding
another format (YAML) would mean duplicating main() or scattering
format branches through every helper.

Add qcom_ptool/spec.py for the canonical internal shape (DiskParams,
PartitionEntry, PartitionsByLun, LoadedSpec) and qcom_ptool/loaders/
with a load(path, image_map) dispatcher keyed on file extension: a new
format becomes a new module plus a suffix registration. Move the conf
parser into loaders/conf.py and apply the image-map override in one
post-load pass, removing the last shared-global coupling.

Behaviour-preserving: every platforms/*/*/partitions.xml is unchanged;
ruff and mypy stay clean.

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
make integration only checks that referenced files exist; it does not
cover size parsing, attribute-bit decoding, LUN grouping or the
image-map override -- the contract any future loader must reproduce.

Add tests/unit/ pinning those cases against conf.load() and the loaders
dispatcher. Wire it in with a make unit-test target, add it to make
check, install python3-pytest in CI, and keep pytest config in
pyproject.toml so pytest runs from the repo root.

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The YAML partition source introduced in [1] needs a YAML parser and a
schema validator at runtime. Declare PyYAML and jsonschema as install
dependencies so pip pulls them in automatically.

No code imports these yet; this only prepares the ground for the YAML
loader. Build, lint, unit tests and the pinned checksum manifest are
unaffected.

[1] qualcomm-linux#124

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The Dependencies section still claimed the tool ran on the standard
library alone. Update it to note the two runtime dependencies now
declared in pyproject.toml (PyYAML and jsonschema) and that pip
installs them automatically.

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Introduce the structured YAML source format from [1] for a single
storage device. loaders/yaml.py loads a `disk` mapping and `partitions`
list, validates it against a packaged JSON Schema, and normalises it
into the same LoadedSpec the .conf loader produces, so both formats emit
byte-identical XML.

Normalisation mirrors loaders/conf.py field for field: defaults are
copied in the same key order, partition_size_in_kb is reused and the
attribute bits are decoded identically. YAML booleans are rendered as
lowercase "true"/"false" so they match the legacy strings. The schema
enforces the YAML footgun mitigations: GUIDs and sizes must be quoted
strings, unknown keys are rejected, and the disk type is constrained to
the known storage classes.

Register .yaml/.yml in the loaders dispatcher behind a late import so
the .conf path never pays for PyYAML or jsonschema, and ship the schema
as package data.

[1] qualcomm-linux#124

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Pin the load-bearing property that the YAML and .conf loaders produce
the same LoadedSpec: an equivalence test against the real
glymur-crd/nvme board, a synthetic multi-LUN spec exercising attribute
bits, and an end-to-end assertion that both formats emit byte-identical
XML through the shared emitter.

Add schema-rejection cases for the documented footguns (unquoted
all-digit GUID, unquoted size, unknown key, missing type-guid,
malformed size, unknown disk type) and a non-mapping top-level
document.

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Introduce the board format from [1]: a board declares a platform and a
list of storage devices, each with partitions. board.py resolves the
three composition mechanisms into a ResolvedBoard - board-level
`extends:` (inherit one base board, override by storage id / partition
name), storage-level `includes:` (concatenate shared _common fragments,
transitively), and variant overlays (--hlos / --boot-fw) applied last.

Merges are deep and field-by-field, matched by stable identifier, with
unmatched entries appended in file order; resolution order is base
board -> derived board -> storage includes -> variant overlays. There
is no partition deletion in v1. Cycles in extends or includes are
rejected.

Input files are validated structurally against board.schema.json (and
include.schema.json for fragments), which stay permissive because
overrides are partial. Each fully-resolved storage is then validated
strictly against the existing partitions.schema.json and reduced to a
LoadedSpec through the same normalisation helpers the YAML loader uses,
so a resolved board emits XML byte-identical to an equivalent
hand-written single-storage file.

[1] qualcomm-linux#124

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Pin the composition contract: extends inheritance with field-by-field
overrides and appends, storage includes with transitivity, variant
overlays applied last (overriding an included partition in place and
appending new ones), the deterministic partition ordering, cycle
rejection for both extends and includes, strict validation of the
resolved storage, and a byte-identical-XML check against the equivalent
single-storage YAML.

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Extend the CLI with --board / --hlos / --boot-fw (and --root) so a whole
board resolves and emits one partitions.xml per storage, in declared
order, from a single invocation. Outputs are given as one -o per
storage; a count mismatch is reported with the storage ids. The board
resolver is imported lazily so the legacy -i single-storage path never
pulls in PyYAML or jsonschema, and that path is unchanged.

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Derived boards and heavy includes make the effective layout impossible
to read from one file. `qcom-ptool show --board <f> [--hlos ..]
[--boot-fw ..]` resolves the full composition chain and prints the
result as YAML, so reviewers can see exactly what will be emitted
without chasing extends/includes by hand.

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Assert that --board emits one XML per storage in order, that an output
count mismatch fails, that exactly one source (-i or --board) is
required, that the legacy single-storage path still works, and that
show prints the resolved spec with variant overlays applied.

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Author the first board definition [1] as a proof of concept:
platforms/boards/glymur-crd.yaml declares both storages (NVMe HLOS +
SPI-NOR boot / platform-config) with every partition inline in the exact
order of the legacy .conf files, and variants/hlos/qcom-deb-images.yaml
supplies the NVMe rootfs size and filename.

Partition order is preserved verbatim because it determines GPT offsets;
reordering changes the emitted layout. That rules out factoring the
shared SPI-NOR blocks through _common includes here, since includes
prepend and the shared groups are interleaved with board-specific
partitions in the current layout - the only order-safe composition for
an existing board is the in-place variant override used for rootfs.

The .conf files are left in place and the Makefile still builds from
them, so generated artifacts and the checksum manifest are unchanged.
The YAML board emits byte-identical partitions.xml and GPT for both
storages, verified against the .conf output.

[1] qualcomm-linux#124

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Resolve the YAML board with the Debian HLOS variant and assert each
storage reduces to exactly the LoadedSpec the corresponding .conf file
produces, which guarantees byte-identical partitions.xml and GPT, and
that storage order is preserved. Guards against drift while both source
forms coexist during the migration.

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Build Glymur-CRD from platforms/boards/glymur-crd.yaml instead of the
legacy per-storage partitions.conf. A grouped rule resolves the board
with the Debian HLOS variant and emits both storage XMLs in one call,
and the board is excluded from the .conf glob. The rule creates the
output directory because a migrated storage need not carry any tracked
file. Generated artifacts and the pinned checksum manifest are
unchanged.

[1] qualcomm-linux#124

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
The YAML loader parity tests referenced platforms/glymur-crd, which is
migrating away from .conf. Add a self-contained .conf fixture under
tests/unit/data/ and point the parity tests at it, and reduce the
Glymur-CRD guard test to a structural check - byte fidelity is covered
by the pinned checksum manifest.

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
Glymur-CRD is now built from its YAML board through the Makefile, so
remove the two legacy partitions.conf files. Generated artifacts and the
checksum manifest are unchanged.

[1] qualcomm-linux#124

Signed-off-by: Igor Opaniuk <igor.opaniuk@oss.qualcomm.com>
@igoropaniuk

Igor Opaniuk (igoropaniuk) commented Jul 1, 2026

Copy link
Copy Markdown
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant